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AE^RACT 

The Portland public school S7stem is decentralized 
and has three administrative areas^ each headed by a superintendent. 
Each area is provided with an evaluation administrator^ and a central 
evaluation ^administrator ^ and- a central evaluation department is also 
maintained. Area evaluation is regarded as an essential function 
which each superintendent maintains for the improvement of programs 
in his area. Central evaluation audits the relative effectiveness of 
area programs and implements the district superintendent's ultimate 
responsibility for the quality of programs. Model policies for 
evaluation have been established in three fields in which central and 
area evaluators have encountered prob;Lems that heeded to be resolved. 
(Author) 



ERIC 



* 
* 

t *** : 



Documents acquired by ERIC include many informal unpublished 

materials not available from 'other sources. EPIC Dakes every effort * 

to obtain the best cop available, nevertheless^ items of marginal * 

reproducibility are oiten encountered and t^is affects the quality * 

of the microfiche and hardcopy reproductions EBIC makes available * 

via the ERIC Document Reproduction Service (EDPS). EDR5 is not ^ * 

responsible for the quality of the original document. Eeproduct ions * 

supplied by EDFS are the best that can be made from the original. * 



U S OEPARTMENTOF HEALTH 
gOUCATlON * WELFARE 
NATIONAL INSTITMTEOF 
EDUCATION - 

TH" DOruVtN^ HAS BEfN »fcPWO 

-Ht PF^SON OWoAN'ZAT-ON OR'OiN 
a:n.,i- points op ViFA Oft OPIN ONi» 

r.ui NATIONAL INSTITUTE 
i , - ' f'OS'TiON 0« PC. C 



Framework of Organization and Policy 
for 

Evaluation in a-Decentralized School District 

By • 

Dr. Victor W. Doherty 
Portland Public Schools 
631 N.E. Clackamas Street 
Portland, Oregon 
9720a 



-ID 



O 
O 

o 



Presented at the Annual Meeting of the American Educational Research 
Association, Washington, D.C,, April 1, 1975* 



Because of the trend toward decentralization of large school systems, 
and the» organizational problf^ias that attend decentralization, it sftould be 
useful to report on the organizational and policy framework for evaluation 
developed in one such district. 

Organizational Framework 

Allocation of evaluation functions in the Portland School District 
reflects the basic policies of the decentralized organization. Responsibility 
for planning, operating, and evaluating instructional programs rests with the 
three decentralized subdistrict superintendents. These functions are dele- 
gated to create as much autonomy, diversity, and competitive creativity with- 
in the District as possible, while maintaining administrative units of suffi- 
cient size that each will still have resources to perform these essential 
functions effectively. Each decentralized subdistrict has an evaluation ad- 
ministrator on its own staff, with responsibility to provide the direct eval- 
uation of instructional programs. 

On the other hand, the Superintendent ''^iif Schools assumes ultimate 
accountability for the success of all programs developed and implemented 
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within each decentralized subdistrict and exetclses that accountability 
I 

by monitoring independently the effectiveness of the programs maintained 
by them. The Central Evaluation Department carries out this function. 

The Central Evaluation Department develops information systems and 
auditing programs that provide information about the functioning of all 
important programs and operations of the subdistricts. It also establishes 
standards and procedures to insure that evaluats^-on done by subdistricts 
is useful and interpretable to the Superintendent. The function of the sub- 
district evaluation administrators is to directly evaluate the efficiency 
and effectiveness of instructional programs of the schools. Finally, it 
might be noted that the principle of decentralizatioa that governs the re- 
lationship between the Central and Area Evaluation Departments is to some 
extent repeated in the relationship between the Area Evaluation Office and 
school principals* Each subdistrict has a program which permits schools 
to apply for funds for instructional improvement, a^M as a part of the 
application, school principals must specify the manner in whicji the improve- 
ment is to be evaluated. Thus, the principal himself is charged with an 
evaluation responsibility, and the subdistrict evaluation administrator 
must provide stahdards and guideliees that will insure that these evaluations 
are useful and interpretable to the subdistrict and district superintendents. 

One special problem that surfaced in this decentralized evaluation pro- 
gram is the difficulty of achieving data comparability where different testing 
programs are maintained by each subdistrict. This became a matter of concern 
in analysis of achievement test results, and also where either District or 
specially funded projects involve more than one subdistrict and corapai;ative 



performance between subdistricts must be assessed. Since each sub- 
district maintains its own testing program using different tests and 
different types of test scores, it became evident that a minimum uniform 
testing program that provides comparable data across subdistricts is 
required. It also became evident that any experimental program or new 
project that involves more than one subdistrict, in which the goals are' 
the same and the need to evaluate is uniform, requires a procedure for 
establishing agreement on the goals of such programs and upon com^n 
measurement procedures and instrumentation. 

The following policies were arrived at with respect to a city-wide 
testing program and cross-Area program evaluations. Some of the city-wide 
testing policies, it will be noted, address matters that are not restricted 
to decentralized district's. 

V 

Tescing Program Policies 

The policies agreed on between the Superintendent, subdistrict 
superintendents, and central and subdistrict evaluators that relate to 
^ testing are basically as follows: 

1. City-wide tests will be administered in the fall of each year 

grades 4, 8, and 11. These tests initially will be limited to basic skills, 
with areas of basic skills designated by the central and area curriculum per- 
sonnel of the Distric^. 

2. Subdistricts will maintain additional testing programs as required to 
satisfy their respective evaluation requirements. 

3 De»elopmental work will continue with the Rasch Model to determine if 
the following capabilities can be provided: 
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(a) Ability to test students in the same giade with different test 
forms to assure testing appropriate to each student's developmental 
level, but still permit combining and averaging of Rasch scale scores 
for research and evaluation purposes. 

(b) Ability to compare results of achievement across different testing 
programs within the District ( by linking items, a part of the Rasch 
procedure) . 

(c) Ability to compare results of achievement among school districts ,^ 
which would cooperate in administering linl^ng ttems and pooling item 

t information so all could relate to a common scale. 
Note: To -the extent the above three capabilities are proved, each 
will be incorporated into the official testing policies and 
processe^s of the District. ^ 
4. Standards of performance will be set for basic skills tests (at, grades 

4, 8^ and 11 initially), to identify students needing additional work in 
these skills. 

5. llie city -wide testing program will operate on an all-student rather than 
sampling basis. Mean test results are needed for each class in each subject, 
and sampling would not permit this. If student sampling were attempted, com- 
plete class data could not be provided each teacher, which could adversely 
affect teacher motivation to test and student motivation to perform. 

'"^ In addition to these basic policies, central and subdistrict evaluators 
have agreed on the following directions in testing: 

1. Development of a goal- item bank which will permit storage and retrieval 

of goals (learning outcome statements) and test items that have been specifically 



designed to measure these goals. Items will be Rasch-calibrated when 
possible. Ultimately, it niay be possible from this bank to put tests ^ 
together for whatever goals are selected and to secure Rasch score estimates 
of total test performance on the items used. As the bank develops, capa- 
bilities for measurement will be extended into all areas of instruction. 
2* Measurement at the high school level will be shifted from general tests 
of academic development to tests that are specific to courses offered. 
Specific course tests, based upon identification of goals by cross-subdistrict 
curriculum committees, will provide a basis for course an^ program evaluation 
(where content of such courses in |ommon to 4 or more high schools), and 
the goal-item bank"^ referred to above will be a resource for developing tests 
aid securing evaluative data for all other courses. 

Cross-Sub^istrict Program Evaluations 

Policies relating to evaluation of programs that are common to twox^r 
all three subdis tricts , and for which the Superintendent desires to have 
comparable data are as follows: 

(a) Method of identifying common goals across subdistricts . Joint 
committees representing each subdistrict shall carry /out the identifi- 
cation cf coninon goals. Membership shall be based oh recommendations 
of *:he three curriculum administrators, and the committees* work should 
be supervised by a curriculum administSrator or specialist agr^d on by 

V, che three curriculum administrators. 

(b) Method of selecting instruments . Instruments shall be selected 

by the same curriculum committee that identified the common goals vfsxng 
criteria developed by the Central Evaluation Department and approved by 



the subdistrict evaluation administrators. The coinnittee should 
receive technical instruction and support from a cefttral and/or 
subdistrict evaluation admini sera tor. 

(c) Type of instrumentation . Selection of the^type of test will be 
in terms of the purpose of the evaluat^pn. Summative evaluation of 
new or experimental programs will usually require a norm-referenced 
test that yields a total test or set of sub-test scores, prefereably 
standardized locally to insure better knowledge of the norming popula- 
tion. Formative evaluation will usually require a criterion-referenced 
test that yields information on specific goals. The basic element in • 
the selection in either case is an analysis of the validity af the 
test under consideration fn terms of the goals of the program. 

(d) T ype of cost mea5;ure . Cost flfeasu^pes shall separately identify 
operating costs (including personn||l and supplies), amortized equipment^^ 
costs, and amotrized capital outlay costs. All costs shall be expressed 
both as total aijd- per pupil. 

(e) Type of score . formative test data shall be expressed In terms of 
raw scores or standard scores where ^valuations involve comparison groups 
Criterion referenced data shall be reported as percentages of students 

0 IP 

currently answering each item (group) percentage of items correct in 
relation to total possible for each goal (group o| individual). - 

(f ) T ype of evaluation and balance (form ative, summative) . evaluation 
of new or experimental programs shall stress sunmative evaluation. Such 
evaluation shall, however, be accompanied by such d^^ta , descriptive of 
program and process as may be needed to interpret the success of lack of 
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the program, and to make possible its replicaf.lon, if desired, 
to produce similar results. On-course correction resulting from 
formative evaluations must be included in such descriptions, 

•r 

(g) Format of report . The format of the report shall consist of 
the following points: 

1. General (introductory) description cif the new or experimentJ^l 
program, its rational^, methods, materials. 

2. 1 A statement of the goals of the program. <2> 

3. A statement of the questions cvaluati"»n is supposed to answer. 

4. A statement describing the evaluation methodology used in 
securing answers to each" question. 

^ 5, Evaluation of ^ogram implementation — data on management, 
suppoi&t , ins true t ion . 

6. Summative evaluation data. 

7. Analysis of costs. 

8. Conclusions 

9. Recommendations 

That subdistricts be encouraged to supplement Central evaluation of 
programs that extend across more than one subdistrict, but that plans for 
such supplemental study be checked out with central evaluation to insure 
proper coordination and procedure. 



Summary Remarks 

^ Ev^,luation capabilities are provided each subdistrict so that the total 
functions of management, including evaluation, are under the direction of the 
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the subdistrict superintendent. Ihls helps to assure the autonomy and 
organizational integrity of the decentralized subdistrict. By controlling 
evaluation of its own progra^^g, each subdistrict controls the most essential 
tool management possesses for insuring its own quality and proper functioning. • 

Central evaluation plays the role of tapping into all programs 'of 
tnanagement;, support (including subdistrict 'evaluation) , and instruction to 
determine if these programs iare producing the outcomes they exist to produce. 
This is based on the assumption that final responsibility for the success of 
^all programs rests with the District superintendent, and that central auditing 
* is required to determine the relative success of the three autonomous sub- 
districts. 

The preceding description of the organizational and policy contexts in.^- 
which central and subdistrict evaluators fulfill their mutually supportive > 
responsibilities can serve as a model for decentralization of the evaluation 
function. 

Completing the organizational framework is the informal, but very 
essential ''Joint Curriculum- Evaluation Comnittee," comprised of subdistrict 
and district evaluation personnel and subdistrict and district curriculum 
personnel. The size of this group is manageable, since there is no central 
supervising staff. Each of the three areas has a curriculum adninistrator 
and they are joined by the central assistant superintendent who coordinates 
any joint curriculiM planning required in the District. Meetings of this group „ 
are h^ld monthly, and they are invaluable in coordinating the work of curriculum 
and ^evaluation. 
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